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ABSTRACT 

Evaluators usually assume that the data they receive 
are valid, or at least, that the data reflect that which is being 
evaluated. In a recent evaluation in a large southern school 
district, evaluators compared their raw interview notes with 
interview notes edited by the teachers interviewed or project staff. 
The program evaluated was Project SEED, an additional mathematics 
class for students of low socioeconomic status. In the present study, 
24 graduate students with no knowledge of the background compared 14 
original and edited pairs of comments/notes in terms of whether the 
edited notes: (l) contained additional information; (2) were more 
positive in tone and intent; (3) were less concrete; and (4) changed 
meaning or content. Edited responses contained additional information 
and were more positive about the project, in the opinion of naive 
raters. Recommendations are made to control the collection of data so 
that the evaluation will be useful to decision makers. Allowing those 
who are being evaluated to control and possibly modify the data 
raises the danger of bias. The raw and edited responses, the survey 
form administered to the graduate student raters, and a table of the 
results are provided. (SLD) 
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HOW PROGRAM STAFF CAN PROVIDE FALSE EVALUATION DATA: 

THE PROJECT SEED EXPERIENCE 

Keith McNeil and Charles Blanchard New Mexico State University 

Objective: The paper will empirically demonstrate how one project 
staff provided false evaluation data. Also included are several 
hints as to how evaluators could avoid this type of situation in 
the future. 

Perspective: Evaluators usually assume that the data they receive 
are valid, or at leaat that the data reflect that being evaluated. 
In a recent evaluation in a large Southern school district 
evaluators had an opportunity to compare "raw interview notes" with 
"edited interview notee." The raw interview notes were edited by 
project staff and ware judged by the evaluatore to be so very 
different from the raw notee that the interview data wae basically 
not included in the final report. The present etudy ussd ratsrs 
who had no knowledge of the situation and hence had no veeted 
interest in the outcome to determine 1f the edited responses were 
Indeed different. 

Techniques: The project being evaluated waa called "Project SEED," 
a 45 minute per day additional mathematics class for low SES 
studente. The curriculum of SEED consisted primarily of logarithms 
and axponsnts and was preeented by "mathematics experts," few of 
whom had teaching degreee. Consequently, the regular teachers had 
to remain in the classroom, performing only perfunctory classroom 
dutlee during that 45-m1nute period. Thus was the program not only 
costly, but focussd on content that wae neither on the district- 
wide teet nor on the list of state competencies. Never the less, 
the project director had convinced the school board and the 
Superintendent that SEED was a good program. Indsed, at the 
beginning of the evaluation, the Superintendent distributed a memo 
to his staff wherein he stated that the SEED project waa the "best 
program 1n my 29 years of educational experience" (Superintendent 
memo in McNeil and Blanchard, 1989). 

Because the project wae being implemented in six schools that 
were under a court desegregation order, the judge ordered that the 
project be evaluated. The project director's numerous attempts to 
control or destroy the evaluation have been pressntsd elsewhere 
(McNeil and Blanchard, 1989) . The present paper focuses on one 
aspect of the evaluation — interviews of the SEED teachers. 

Method: A plan for interviewing the teachers was prepared and was 
shared with the project director. The plan specified a random 
sample, but the project director wanted to construct his own 
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sample, and balked at our plana to sample by saying that we might 
obtain 1n our sample only the new teachers, those with the leaet 
experience, etc. He waa requested to provide the relevant 
varlablee and the scoree of each teacher on those variables so ths 
sample could be et rati fled on thoae variables. He refused, and 
pulled his political strings such that his N sampls M was the one 
that waa finally ueed. 

Two evaluatore (the first author and his boss) Interviewed 
three teachere each with the atructured Interview. The 
interviews re wrote down the responses and read them back to the 
teachere to make euro that they had accurately captured the essence 
of the response. Once the notee were typed and proofed by the 
evaluators, ths typed notee were tranemltted to the project 
director with the requeet that the notee be edited by the teachere. 
Edited they were, ae Indicated 1n Appendix A. The evaluatore 
received what we described ae "highly edited" notee. The original 
notee were not returned (fortunately copies had been kept). Each 
of the teacher's notss wsrs totally rstyped with the earns machlns. 
Ths ©valuator's conception of editing for clarification or grammar 
was Implemented by the project etaff 1n what can only be deecHed 
ae a "snow job." Xn many casss the edited commente were judged by 
the evaluators to be totally unrelated to the original commente. 
In other caaee, the edited commente were eeen ae depicting the 
project 1n a much more favorable light. Since the evaluatore felt 
that the project etaff (actually most likely the Project Director) 
did not cooperate with the Intent of the Interview effort, the 
evaluators did not fsel obliged to report theeo reeulte 1n detail. 
Consequently, the entire effort waa eummarized 1n the final report 
as: "The six SEED tsachere Identified by SEED management were 
extremely posltivs about SEED and thsir rols 1n the project. In 
terms of the students of ths project, SEED teachere maintained that 
students gained not only additional math knowledge, but also 
increased sslf sstsem. The major weakness Identified by the SEED 
teachers was ths constraint of each student taking only one 
semester each year." (McNeil, 1987) 

Rating Data: In the present study raters who were unaware of the 
background situation, ratsd 14 pairs of commente. Out of ths 30 
comments (6 SEED tsachsrs times 5 questions), 16 comments were 
omitted for one of the following two reasons: 1) comments were so 
different that they couldn't be compared, or 2) comments rsquired 
too much knowledge of the project or setting. 

Each rater rated the original and edited pairs of comments on 
each of four scales. (See Appendix B for an example of the rating 
form. ) Thess scales wsrs chosen because they were suspected to be 
the major ways that the edited responses differed from the 
original : 

1. contains additional information. 

2. more posltivs in tone and Intent. 

3. less concrsts. 

4. changes meaning or intent. 



Results: The ratings were obtained from 24 graduate students who 
were unaware of the background of the project and Ite attempts to 
avoid being evaluated. The aprlorl rating dlfferencee between the 
original and edited verelone were teeted with the one-tailed 
correlated t test, because of the aprlorl hypotheses. Tab Is 1 
contains the reeulte, summed acroee the 14 1teme, ae well as 
summary information for ths Individual 14 reeponeee. 
Question 1. Our hypothesis that the edited ratings contained 
additional Information waa supported in 5 out of the 14 
comparisons. When the reeulte were summed over the 14 items, the 
hypothesis was also supported. Thue naive ratere felt that the 
edited reeponeee did contain additional content, ae compared to the 
original responses. 

Question 2. Our hypothesis that ths edited ratings were more 
positive 1n their tone and Intent waa supported 1n 7 out of the 14 
compeHeone. When the reeulte were summed over the 14 comparieone, 
the hypothec 1s was aleo supported. Thue, our Initial and possibly 
emotionally biased reaction that the edited reeponeee caet the SEED 
project In a more positive light than did the raw responeee was 
upheld by the naive ratere. 

Question 3. Our hypothec 1e that the edited ratings were less 
concrete wee not eupported. In looking at the reeulte for each of 
the 14 comparieone, we noted that queetion 3 tended to be rated in 
much the earns way as the flret two queetione. While the raters 
were cautioned to read each comparison and sach question carefully, 
we have no assurance that the ratere carefully responded to the 
taek. Queetione 1 and 2 may well have sst up a reeponee bias in 
how questions 3 and 4 were rated. 

Queetion 4. Our hypothesis that the edited ratings changed the 
meaning or intent wae eupported in 2 of the 14 comparieone. With 
hindsight, this was not a vary good question to aek naive raters, 
becauee the raw and sditsd responses did, in a general sense, 
provide similar information. Only someone familiar with the entire 
situation could realize the ramifications of some of ths small 
changss . Furthermore, we eliminated from the rating form those 
comparisons that changed the meaning so much that we felt raters 
could not make meaningful ratings. 

What was rsally different in the raw and edited ratings was 
the subtle message that ths projsct was successful and that thsre 
were no problems within the project. Not only did the edited 
responsss contain additional information, as judgsd by naive 
ratsrs, but thess raters also bslisvsd that the edited responses 
wsrs mors positlvs about ths SEED projsct. 

Recommendations: Due to the experiences of evaluating SEED, and 
the supporting evidence provided by the raters, the following 
recommendations ars made: 

1. Do not allow the data gathering process out of sight of the 
evaluators. 

2. Hake clear to thoee being evaluates why certain evaluation 
procees ars bsing followsd. When projects rseist being evaluated, 
try to explain the benefits of being evaluated and the consequences 



of not being evaluated. 

3. Make aura that tha evaluation haa tha aupport all tha way up 
tha admin 1etrat1ve ladder. In th1a caaa, even though the project 
was ordered by a Federal court to be evaluated, the project 
director had the ear of tha superintendent and eucceeafully fought 
the evaluation all the way. 

4. Report any tampering of data when it occura; hopefully the 
organization will deal with the issue. 

6. If the organization cannot deal with the 1eaue, or refuaea to 
deal with the issue, then there ehould be a central clearing place 
to report theee flndlnge, aa dlacueaed by Slavin (1990). 
6. If all avenuea are blocked, then live with the problem until 
you get out of the organization, and than write a conference paper 
(or two depending upon the clrcumstancea). 

Cone lua ion: In order for evaluation data to be ueeful to dedal on 
makere, the data must be valid. Allowing thoee who are being 
evaluated to take control of the data and poaeibly modify the data 
may reeult in the data becoming blaeed in a more poeitive 
direction. If data are euapected of being manipulated, then the 
evaluator haa the right, Indeed the obligation, to either report 
the tampering or to at leaet not report thoee data. 

Epilogue: 

1. A five pereon team wae evaluating the entire operation of 
six schools. The two pereons who had reeponeibi Hty for evaluating 
the SEED project were removed from that evaluation team for the 
subsequent year. 

2. SEED haa not been evaluated in that Diatrlct in the 
succeeding three year 8. 

3. SEED wae given a five-year contract to continue operations 
In the Dietrict during the laet Board Meeting that the 
Superintendent attended, before he left the Dietrict. 

4. The SEED budget and effort haa incrwaaed aa the curriculum 
is now in some of the middle schools. 

5. SEED was also asked to explore Implementing their 
techniques in ths Language Arts curriculum. 

6. Ths leader of the evaluation team is no longer in the 
District. 
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APPENDIX A: THE 14 RAW AND EDITED RESPONSES 
RAW RESPONSES 

1. Because teaching it all 3 years can hava a different curriculum 

over the three years, leading to more sophisticated 

curriculum. 

SEED tsachsrs know the regular teachers real well because they 
have been there two previoue yeare for half a eemeeter. 

2. Poeitive Impact - teachee habits of succsss throughout Ufa 
Critical thinking skills 

Lessen the feare etudente have to articulate an opinion 

3. The Weet Dallae Centere were prepared for the project. 

Weet Dallae center etudente need their eelf image improved 
The teaching methodology that is different 1e an advantage 

4. Not enough money for Increased salaries, to put some stability 
(e.g., retirement). Too much driving. 

5. Assuming thsre are weakneeeee, we can't epend more time with 

etudente. Because some etudente are only taught for a 
eemeeter. 

6. Increaeed funding to allow SEED classes to go the entire year 
would be deeirable. 

7. Scheduling could improve driving problems, but probably not 
eince they had all claeeee to choose from the firet eemeeter. 

8. The option to continue with some classes ae suggeeted by 

teacher. 

Recognizing we are not perfect, but continue to Improve. 

9. Have more than 45 minutee with each claee. 

10 math content — Very effective — curriculum is geared to see the 

big picture of mathematics; see connections between all these 
different elements. Dssper understanding of fractions. 

11 Would want students to generalize the preciseneee (i.e., 
above, on top of, read entire problem and look for operation 
instead of automatically adding). 

12. Hope they develop a level of confidence to ask questions about 
other sub j sets. 

Want to participate and become involved in other subjects. 
They will know that they can. 

13 If they feel good about themselves they'll want to do well. 
Take fear away from asking questions. Desensitize students 
to questions. 

14 Thsir ability to speak. Curriculum - goee from one-word 
answers to why. 

Studente will be able to give a clear response and feel good 
about learning. 
EDITED RESPONSES 

1. A three ssmester exposure allows students to discuss 
sophisticated topics in mathematics. Tha partitioning of our 
curriculum. Assigning different topics to each grade, allows 
students to review material in an exciting new context. 
The SEED spsciallsts know the regular teachers from observing 
classss in prsvious ysars. This helps foster better 
communication between the regular teacher and the SEED 
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special let. 

2. Project SEED teaches habits of success, thsrsby giving thsm 

confidence they nssd to carry thsm through high school and 
col logs and into csrssra worthy of thslr ability. Projsct 
SEED'S teaching methods Improve students' critical thinking 
sMIIe. These methods also leeaen the fear atudents may have 
to articulate. 

3. The professionalism of ths Projsct SEED mathematicians 
The group discovery (Socratlc) methodology uaed. 

The high level of the mathematics which 1a taught and 
continually tailored to the needs of ths claaa. 
The above combination (mathematicians, discovery teaching, 
high- level mathematics) provides ths baa la for the enhancement 
of the etudent's eel f- Image. 

4. Weaknesses of Project SEED 1n the Learning Centers, Hks those 

elsewhere, arise basically from shortags of money: low 
sslsHss and not enough senee of stability (e.g., no 
retirement), long houra and too much driving 

5. studente srs limited to only one to only one eemeeter of 
Project SEED per academic year. 

6. Increased funding to si low SEED to go ths sntlrs yssr. 

7. Conditions would Improvs 1f one could schedule classes to cut 

down driving. Since, however, 1n the second semester we hsvs 
to tssch exactly those claesee which were not taught 1n the 
fall, scheduling will be, 1f anything, more difficult. 

8. Occasional ly, a teacher will requeet an extension of the SEED 

claae which cannot be granted due to scheduling constraints. 
Overall, the effectiveneee of the SEED program 1s Improved by 
regularly resisting stagnation in the mldet of an on-going 
queet for excellence. 

9. Students could be expossd to SEED more times psr class period 

or more claee periods par year. 

10. math content — Very effective — curriculum 1s geared to ehow the 

big picture of mathematlce; it illustratee connections between 
many different elements. A deeper underetandlng of fractions 
1s one important benefit. 

11. X would want atudents to extend their 1nqu1s1tivsnees and 

curiosity to othsr subjects; not to be afraid of advancing and 
defending their own ideas and to use precision in sxprsssing 
themselves. 

12. I would hope that as a result of the SEED experience, students 

would have acquired an increased hunger for knowledge and the 
courage and pereeverance to actively seek such knowledge, 

regardless of any risks or obstacles that might otherwise 
hinder this pursuit. 

13. If thsy fssl good about themselves, they will want to do well 

in all sub j sets. 

Ws desensitize students to questions. We reduce the anxiety 
associatsd with answsring qusstions. 

14. Ths objsctivs would bs for ths studsnts to not only speak 

precissly and cogently, as a rssult of thinking clearly and 
logically, but to fssl good about acquiring such skills. 

6 

ERIC 8 



APPENDIX B RATING FORM 



SEED RATINGS 



A special math project axiata 1n Dallas Texas, callad "Proj*xt 
SEED. " Ths projsct provldaa 46 minutss of aupplamantary a>ru,h 
instruction to atudanta in grades 4, 5, and 6 in thraa schools 1n 
Wsst Dallas and thraa schools in South Dallaa. Thaaa schools srs 
in low socioeconomic araaa of ths city. Tha taachsrs in ths 
projsct ars not regular smploysss of ths district, but sra full- 
tims smployees of ths company that haa tha contract to conduct 
Projact SEED. SEED ia an acronym for Supplementary Early Education 
for tha Disadvantaged. 

Survay raaponaaa were obtained from tha asms teachers on two 
diffsrsnt occssions. Ws nssd your hslp in comparing thsss 
rssponsss. Plssss rssd ths survsy qusstion and the pair of 
raaponaaa. The atatsmsnt on ths Isft and ths statsment on tha 
right are the two raaponaaa to that qusstion by one teacher. You 
are to compars these response on ths baa is of the four attributes 
that ara at tha bottom of tha paga. Plsass circle tha rating which 
most reflacts your raactlon to the atatamant comparleon. 

We thank you for participating and will provida tha rssult of 
this study back to you. 



PImm read the following pair of reeponsee. Tha atatamant on tha 
laft and tha atatamant on tha right ara two raaponaaa from tha same 
taachar. You ara to compara thaaa raaponaaa on tha baa 1a of tha 
four quaatlona balow. Plaaaa circle your raaction to tha atatamant 

comparlaon. 

WHAT ARE THE WEAKNESSES OF PROJECT SEED IN THE LEARNING CENTERS? 



((EDITED)) 

8. Occasionally, a taachar will 
raquaat an ext melon of tha 
SEED class which cannot ba 
grantad dua to achaduling 
constraints. Ovarall, tha 
affactl vanaaa of tha SEED 
program 1a aupportad by 
ragularly raa 1 at 1ng stagnation 
1n tha midst of an on-going 
quaat for axes 1 lanes. 



((RAW)) 

Tha option to continue with soma 
claaaaa aa auggmstad by taachar. 

rscognizing that wa ara not 
parfeet, but continua to 
Improva. 



1. Comparad to tha atatamant on tha laft . tha atatamant on tha right 
contains additional information. 

Strongly Disagraa Diaagraa Nautral Agrsa Strongly Agraa 



2. Comparad to tha atatamant on tha laft . the atatamant on tha right 
ia mora poaitive in it'a tona and intent. 

Strongly Disagree Disagree Neutral Agree Strongly Agree 



3. The statement on the right is more concrete than tha atatamant on 

the left. 

Strongly Disagree Disagree Neutral Agree Strongly Agree 



A. The statement on the right changes the meaning or intent of the 
statement on the left . 

Strongly Disagree Disagree Neutral Agree Strongly Agree 
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TABLE 1. RESULTS FOR THE FOUR RATINGS. 



Significant 

Quest 1 on Compar 1 sona Probata 1 1 1 t y 



1. Comparad to tha statement on tha loft . 5/14 .02 
tha statement on tha right contain • 

additional Information. 

2. Comparad to tha atatamant on tha laft . 7/14 .01 
tha atatamant on tha right 1a mora 

positive 1n It's tona and intant. 

3. Tha statement on tha right 1a mora 0/14 .99 
concrata than tha atatamant on tha laft . 

4. Tha atatamant on tha right changee 2/14 .99 
tha maaning or intant of tha atatamant 

on tha laft . 



ABSTRACT 

HOW PROGRAM STAFF CAN PROVIDE FALSE EVALUATION DATA: THE PROJECT SEED 
EXPERIENCE 

Kaith McNeil and Char lea Blanchard Naw Maxico Stata University 

Evaluatora uaually assume that tha data they receive are valid, or 
at least that the data reflect that being evaluated. In a recent 
evaluation in a large Southern echool dietrict evaluatora had an 
opportunity to compare "raw interview notee" with "edited interview 
notea." Tha edited notea were judged by the evaluatore to be ao very 
different from the raw notea that the interview data waa baaically not 
Included in the final report. The preeent atudy uaed ratere who had no 
Knowledge of the situation and hence had no veated interaat in the 
outcome. 

Each rater rated the original and edited pairs of comments on each 
of four scales. These scalee were chosen because they were suspected 
to be the major ways that the edited responses differed from the 
original : 

1. contains additional information. 

2. more positive in tone and intent* 

3. less concrsts. 

4. changea meaning or intent. 

In order for evaluation data to be useful to decision makers, the 
data must be valid. Allowing those who are being evaluated to take 
control of the data and possibly modify the data may reault in the 
data becoming more positive. If data is suspected of being 
manipulated, then the evaluator has the right, indeed the obligation, 
to not report thoss data. 
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